no need for multiple mongodb builds : we have multiple zenko branches, and just one mongodb per zenko branch

both mongo 8.0 and 8.2 are still committed : this is not needed

we don't need to store multiple versions of either of these components in the repo : so we can choose to organize in the exact way we want, and keep this much simpler.

→ choose in which folder we store them, in a consistent fashion, e.g. either store each in a folder which has the same name as the image ; or just /<major_minor> if we want to avoid mismatch in case of upgrade

I removed the version-based folder branching and standardized the Mongo image layout to a single consistent structure per image, which simplifies context resolution and avoids mismatch risks during upgrades.

make deps_key and prefix match, for simplicity?

not correct, this would keep "removed" files

Agreed and fixed. I removed copy-based vendoring instructions because they do not handle upstream deletions safely.

"copying" is intrinsically dangerous and complex, as it

• automatically discards all changes that could have been made to the charts
• starts from scratch each time, without considering the history: i.e. the common ancestor between the "old" and "new" code

These problems are already solved by Git, so may be best to use Git here.
In particular, git subtree [1] allows to easily merge a "remote" repository in a remote one. This kind of approach would mean we don't copy and diff, but instead perform a proper git merge.

The drawback is that this command is probably not as common for everyone ; but the specific usage can be documented here, and/or wrapped in a makefile (similar to what is done for mongo chart)

Going in to more details, there is an extra subtlety here because we don't want the full repo, but only a subset. There are solutions all over the internet, but it requires a few more commands: c.f. [2]

# Add the remote repository
$ git remote add bitnami-containers https://github.com/bitnami/containers.git
$ git fetch bitnami-containers

# Create a branch containing only the subfolders of the remote repo we are interested in
$ git checkout bitnami-containers/main
$ git subtree split -b bitnami-upstream -P bitnami/mongodb-sharded/8.2/debian-12 -P bitnami/mongodb-exporter/0/debian-12

# Back to our "dev" branch, merge the subtree
$ git checkout -b improvement/ZENKO-xxx

# First time: just "add" the subtree, without committing every change
$ git subtree add -P solution-base/images bitnami-upstream --squash

# Next times: merge upstream, to consider both our changes & bitnamis. Not sure we will want to squash changes (if we see only relevant changes, may be nice to see them individually in history), but not the most important thing anyway
$ git subtree merge -P solution-base/images bitnami-upstream --squash

Note:

• git subtree is part of git "contribs", but not installed by default. So need to install it, c.f. [3]

  $ cd $(brew --prefix)/share/git-core/contrib/subtree
  $ make
  $ make prefix=$(brew --prefix)/opt/git install
  

• since bitnami repo is large, git subtree split can be quite slow if the whole history was retrieved. It may be much faster if the whole repository is not fetched, not sure if that works though...
• The sub-path is making git subtree more complex to build the upstream branch or falling back to advanced git commands instead should be considered, c.f. [4] : i.e. basically we use git read-tree the first time, and followup merge upstream with git diff | git apply.

[1] https://manpages.debian.org/testing/git-man/git-subtree.1.en.html
[2] https://medium.com/@icheko/use-a-subfolder-from-a-remote-repo-in-another-repo-using-git-subtree-98046f33ca40
[3] https://codeengineered.com/blog/how-to-install-git-subtree/
[4] https://stackoverflow.com/questions/23937436/add-subdirectory-of-remote-repo-with-git-subtree

Just asked Claude, seems to automate quite well (though the code will need to be reworked to also handle the list of containers to sync in the makefile/build script)

BITNAMI_REMOTE := bitnami-containers
BITNAMI_REPO   := https://github.com/bitnami/containers.git

create-remote:
	@git remote get-url $(BITNAMI_REMOTE) >/dev/null 2>&1 || git remote add $(BITNAMI_REMOTE) $(BITNAMI_REPO)

# Usage: make vendor-sync CHART=mongodb-sharded PREFIX=solution-base/mongodb/charts/mongodb-sharded
vendor-sync: create-remote
	git fetch $(BITNAMI_REMOTE) main --depth=1
	git subtree split --prefix=bitnami/$(CONTAINER) $(BITNAMI_REMOTE)/main -b vendor/$(CONTAINER)
	git subtree merge --prefix=solution-base/images/$(CONTAINER) vendor/$(CONTAINER) --squash

# First-time import variant
vendor-add: create-remote
	git fetch $(BITNAMI_REMOTE) main --depth=1
	git subtree split --prefix=bitnami/$(CHART) $(BITNAMI_REMOTE)/main -b vendor/$(CONTAINER)
	git subtree add --prefix=solution-base/images/$(CONTAINER) vendor/$(CONTAINER) --squash

Then for any container:

make vendor-add CONTAINER=mongodb-sharded

to clarify the intent here : however we do it, we will make changes to the image, but we need to manage it.
Such in (git) history we need to have something like:

* Initial mport from upstream commit XXXX
* Our own tweaks [for exemple: modify makefile to take mongodb version in argument]
* More tweaks [for exemple: bump baseline image hash]
* ...
* Merge upstream commit YYY
* ...

So we need to make sure we actually "merge" upstream with our own changes.
For the charts, we ended using patch files - works, but not very efficient: but we need to do something in this case as well...

I switched documentation and workflow to a Git-based vendoring approach (git subtree) so upstream updates are merged properly and history is preserved.

should be a single ARG (e.g. "MONGODB_VERSION") in the Dockerfile.....whose value we retrieve from the deps.yaml, and is passed to docker build

The Dockerfile now uses one MONGODB_VERSION build arg, and CI passes its value from deps, so version updates are centralized and consistent.

redundant with the doc in images/mongodb-shared/README.md
either keep both here, or just use a "pointer" to the other file here

(and you see the cost of redundancy: you forgot to update this one to match your latest changes)

I removed redundant step-by-step image bump content and left a pointer to the canonical mongo image readme to avoid drift.

also depends on deps.yaml, and possibly other build scripts

I removed the separate path-filtered Mongo workflow and moved Mongo image builds into the main pipeline, so trigger drift on deps/build script changes is no longer an issue

why use another mongodb workflow?

• since builds are cached and the tag is computed, it does not cost anything to rebuild
• path filtering can be tricky and cause unexpected results
• it duplicates trigger conditions, which can easily get out of sync and create inconsistencies (or bugs) in the jobs actually executed
• it makes it harder to see what's going on overall (when opening a PR/...)
• it prevents using the just built image in the same build, as there is no dependency: so the mongodb image used may not have been built

Mongo build is now part of the main CI flow with proper job dependencies, so tests use images built in the same run and overall CI behavior is easier to follow.

this tags is a moving tag (may change on every commit), should probably not be used...

The pipeline now builds and consumes immutable tree-hash-tagged images for Mongo components instead of relying on floating semver tags.

the tag with the "tree-hash" that we built MUST be included in the solution-base ISO (do not depend on a floating tag!) and used when deploying mongo in tests

c.f. how this is done for kafka images (which also have such a tree hash)

I aligned Mongo handling with the Kafka pattern: the tree-hash image tag is propagated into solution-base build inputs and test deployment paths, so both ISO content and tests use the exact built image.

CONTAINER is not an argument:

• we know exactly which values are possible and MUST be used
• merging/... should be done at the exact same point, we don't want to mix-and-match different versions of exporter/mongo/os-shell
• git fetch and subtree split are somewhat costly: we should avoid doing them multiple times

typically this means there should be multiple rules : creating remote, fetching/updating "vendor" branch (split subtree), then one rule for each image

Addressed. I removed free-form CONTAINER/BITNAMI_PREFIX usage and moved all allowed values/ref mappings into fixed Makefile rules.

this does not work today : main does not contain bitnami/mongodb-sharded/8.0 → should we have something else instead of main ?

• may be ok once we move to 8.2, but not just now...
• how does it work when we move from 8.0 to 8.2 ?

mongodb-sharded is now split from a pinned upstream commit that still contains bitnami/mongodb-sharded/8.0/debian-12, while mongodb-exporter and os-shell are still split from upstream main. This is now explicit in Makefile.

same BITNAMI_PREFIX is not an input, the goal of a makefile is to set these values instead of relying on some external knowledge...

Prefixes and refs are now defined in the Makefile itself, not passed from CLI arguments. See Makefile.

outdated

@benzekrimaha

already fixed upstream, should be dropped

@benzekrimaha

kind of unexpected to have just a commit hash here : I know this is needed because the 8.0 images have been removed from the upstream repo since then, but worth adding a comment here to explain this is the "latest" commit on main which has the 8.0 image

I added a comment explaining that the pinned hash is the latest upstream commit known to still contain bitnami/mongodb-sharded/8.0/debian-12.

nit: we are removing the debian-12 "information" from the image tag. Is it because we consider it useless/redundant, or just to simplify script?

Intentional. We keep distro information in the source layout and Dockerfile base digest, and use clean semantic app version tags in deps.yaml for consistency with build args and immutable tree-hash tagging. If preferred, we can restore distro suffixes, but current flow does not rely on them.

should we not fetch whole history?

• so the subtree split can indeed see all the changes that happened on the subtree
• this way we also don't need to fetch specifically the mongodb_sharded ref, and fully rely on the refs set in the variables

I updated Makefile accordingly:

• clarified that main refers to Bitnami upstream by introducing BITNAMI_UPSTREAM_MAIN_REF.
• removed shallow/special-case fetches and now fetch full Bitnami main history once in fetch-remote.
• kept refs centralized in variables (BITNAMI_*_REF) so subtree split fully relies on declared refs.
  This should make subtree split history-complete and remove the ad-hoc extra fetch for mongodb-sharded.

not up to date

same for a few others below

Pattern /mongo/ won't match ghcr.io/scality/zenko/os-shell, so the os-shell image tag won't get the tree hash appended. E2e tests will try to pull os-shell:12 instead of os-shell:12-<tree-hash>. build.sh:70 already uses the correct broader pattern.

The sed pattern /mongo/ doesn't match os-shell, but build.sh:70 uses the broader /scality\/zenko\// pattern. CI pushes all three images with the tree hash suffix (${TAG}-${TREE_HASH}), so kind deployments will try to pull ghcr.io/scality/zenko/os-shell:12 (without the hash) which doesn't exist.